Active Message Applications Programming Interface and Communication Subsystem Organization

High-performance network hardware is advancing, with multi-gigabit link bandwidths and sub-microsec-ond switch latencies. Network-interface hardware also continues to evolve, although the design space re-mains large and diverse. One critical abstraction, a simple, portable, and general-purpose communicationsinterface, is required to make effective use of these increasingly high-performance networks and their ca-pable interfaces. Without a new interface, the software overheads of existing ones will dominate communi-cation costs, and many applications may not benefit from the advancements in network hardware. This document specifies a new active message communications interface for these networks. Its primitives,in essence an instruction set for communications, map efficiently onto underlying network hardware andcompose effectively into higher-level protocols and applications. For high-performance implementations,the interface enables direct application-network interface interactions. In the common case, for applica-tions exhibiting locality in communication, these interactions bypass the operating system. To enable theconstruction of large-scale, general-purpose systems, the interface supports the protected multiprogram-ming of many users onto finite network resources.This document also describes a prototype system thatuses the virtual-memory facilities of the Solaris operating system to implement virtual networks that sup-port protected, network multiprogramming. The system caches the active communication endpoints in net-work-interface memory and demand-pages them between the host and network-interface memories.